In this notebook, we will apply the linear model approach presented in (1_simple_example) to a real-life data set.

# load packages
library(tidyverse)
library(biobroom)
library(MSnbase)
# set up standardised plotting scheme
theme_set(theme_bw(base_size = 20) +
            theme(panel.grid.major=element_blank(),
                  panel.grid.minor=element_blank(),
                  aspect.ratio=1))
cbPalette <- c("#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7", "#999999")

We start by reading in the data. Our input here is the protein-level quantification for the Nocodazole arrest/release experiment conducted for the OOPS NBT paper. In this experiment, we wanted to assess changes in RNA binding in arrested/released cells. To do this, we quantified “total” protein abundance and RNA-bound (extracted by OOPS) protein abundance. The peptide-level abundances have been aggregated to protein level abundance and center-median normalised. Proteins with missing values have been removed.

total_protein_quant <- readRDS("../raw/total_res_pro_agg_norm.rds")
rbp_protein_quant <- readRDS("../raw/rbp_res_pro_agg_norm.rds")

The input data are in MSnSets. As a reminder, the MSnSet class mimics the ExpressionSet class and contains 3 matrices: 1. assay data (obtained via: exprs) 2. feature data (fData) 3. phenotype data (pData)

The assay data is the quantification of the features (PSMs/peptides/proteins) and contains one column per sample

The feature data describes each feature, e.g peptide sequence, master protein accession, retention time etc etc

The phenotype data describes the samples

Let’s take a look at our total protein quantification data. If we “print” the object, we get a summary including the processing steps performed. So, here we have 2761 features (proteins) quantified across 10 samples. The varLabels describe the “Condition”, “Replicate” and “Type” for these samples. We’ll take a look at these in more detail shortly. We can see that there were originall 20171 features which were combined into 18111 features using a user-defined function. This was the step at which peptides with the same sequence but different variable modifications were aggregated. Then, these 18111 features were combined into the 2761 features (peptide->protein aggregation). Finally, the data was center-median normalised and missing values with imputed with knn.

print(total_protein_quant)
MSnSet (storageMode: lockedEnvironment)
assayData: 2761 features, 10 samples 
  element names: exprs 
protocolData: none
phenoData
  sampleNames: Abundance.F1.126.Sample Abundance.F1.127N.Sample ... Abundance.F1.131.Sample (10 total)
  varLabels: Sample_name Condition Replicate Type
  varMetadata: labelDescription
featureData
  featureNames: A0AVT1 A0MZ66 ... Q9Y6Y8 (2761 total)
  fvarLabels: Checked Confidence ... CV.Abundance.F1.131.Sample (47 total)
  fvarMetadata: labelDescription
experimentData: use 'experimentData(object)'
Annotation:  
- - - Processing information - - -
Subset [20171,10][20171,10] Thu Aug  9 16:17:13 2018 
Combined 20171 features into 18111 using user-defined function: Thu Aug  9 16:17:32 2018 
Combined 18111 features into 2761 using user-defined function: Thu Aug  9 16:17:46 2018 
Normalised (center.median): Thu Aug  9 16:17:51 2018 
Data imputation using knn Thu Aug  9 16:17:58 2018 
  Using default parameters 
 MSnbase version: 2.4.2

Here’s the top of the assay data

print(dim(exprs(total_protein_quant)))
[1] 2761   10
print(head(exprs(total_protein_quant), 2))
       Abundance.F1.126.Sample Abundance.F1.127N.Sample Abundance.F1.127C.Sample Abundance.F1.128N.Sample
A0AVT1                5.387263                 5.379489                 5.296457                 5.356980
A0MZ66                5.018771                 4.926415                 4.876517                 4.955341
       Abundance.F1.128C.Sample Abundance.F1.129N.Sample Abundance.F1.129C.Sample Abundance.F1.130N.Sample
A0AVT1                 5.444870                 5.532511                 5.411776                 5.353822
A0MZ66                 4.735426                 4.886217                 4.834322                 4.645590
       Abundance.F1.130C.Sample Abundance.F1.131.Sample
A0AVT1                 5.417480                5.350928
A0MZ66                 4.830086                4.634653

… and the associated feature data. Notice that there are many columns in the feature data. These are all the additional columns output from PD in addition to the quantification. They are all stored here in case they are required.

print(head(fData(total_protein_quant), 2))

… and here is the phenotype data. As we can see, we have 3 replicates each of “M”, “G1” and “S” phase, plus an additional Control sample. For our purposes, we’re only going to be interested in the M and G1 phases so we can remove the other data. Both the total and RBP quantification objects have the exact same order

print(pData(total_protein_quant))

Below, we make a boolean from Conditions to subset the samples to those which are “M” or “G1”

# identify the samples we want to keep
samples_to_keep <- pData(total_protein_quant)$Condition %in% c("M", "G1")
print(samples_to_keep)
 [1] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE FALSE FALSE FALSE
total_protein_quant <- total_protein_quant[, samples_to_keep] 
rbp_protein_quant <- rbp_protein_quant[, samples_to_keep]
print(pData(total_protein_quant))
print(pData(rbp_protein_quant))

To detect changes in RNA binding, we can only consider RBPs where we have also quantified the total protein. Below, we identify these cases by intersecting the rownames of each MSnSet (the protein names)

intersecting_proteins <- intersect(rownames(total_protein_quant), rownames(rbp_protein_quant))
cat(sprintf("Out of a total of %s RBPs quantified,\nwe have total protein quantification for %s proteins = %s %%",
            length(rownames(rbp_protein_quant)),
            length(intersecting_proteins),
            round(100*length(intersecting_proteins)/length(rownames(rbp_protein_quant)), 2)))
Out of a total of 2149 RBPs quantified,
we have total protein quantification for 1916 proteins = 89.16 %

Below, we convert the MSnSet into a “tidy” format data.frame using tidy

total_exprs <- total_protein_quant[intersecting_proteins,] %>% # subset to intersecting proteins
  tidy(addPheno=TRUE)  %>% # "tidy" the object, e.g make it into a tidy data format --> long
  mutate(intensity=value) %>% dplyr::select(-value) # rename the "value" column -> "intensity"

Top of the total protein expression data.frame. See how each intensity value now has it’s own row with the other columns describing the associated aspects of the intensity value, e.g the protein and experimental condition

print(head(total_exprs))

Now we do the same for the RBP quantification and then concatenate the two data frames together.

oops_exprs <- rbp_protein_quant[intersecting_proteins,] %>%
  tidy(addPheno=TRUE)  %>%
  mutate(intensity=value) %>% dplyr::select(-value)
combined_exprs <- rbind(total_exprs, oops_exprs)

We want to tell R which is the order of the values in the condition and type columns so that the fold changes are in the expected direction, e.g positive = higher in G1 vs M.

combined_exprs$condition <- factor(combined_exprs$Condition, levels=c("M", "G1"))
combined_exprs$type <- factor(combined_exprs$Type, levels=c("Total", "OOPS"))

Now we model the protein intensity according to the models described in 1_simple_example_vd.Rmd. As an example, let’s see the results from just applying the models to a single UniprotID.

combined_exprs %>% filter(protein == 'A0AVT1') %>%
  ggplot(aes(Condition, intensity, group=1, colour=factor(Sample_name))) +
  geom_point(size=2) +
  stat_summary(geom="line") +
  scale_colour_discrete(name="Tag") +
  xlab("") +
  facet_wrap(~type)

fit <- combined_exprs %>% filter(protein == 'A0AVT1') %>%
  lm(formula=intensity~Condition*Type)
print(summary(fit))

Call:
lm(formula = intensity ~ Condition * Type, data = .)

Residuals:
      Min        1Q    Median        3Q       Max 
-0.087537 -0.048708  0.007263  0.041451  0.069459 

Coefficients:
                     Estimate Std. Error t value Pr(>|t|)    
(Intercept)           6.53699    0.03620 180.590 9.89e-16 ***
ConditionM           -0.76710    0.05119 -14.985 3.88e-07 ***
TypeTotal            -1.07394    0.05119 -20.979 2.80e-08 ***
ConditionM:TypeTotal  0.64836    0.07240   8.956 1.92e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.0627 on 8 degrees of freedom
Multiple R-squared:  0.988, Adjusted R-squared:  0.9835 
F-statistic: 219.6 on 3 and 8 DF,  p-value: 5.07e-08
fit <- combined_exprs %>% filter(protein == 'A0AVT1') %>%
  lm(formula=intensity~Condition*Type+Sample_name)
print(summary(fit))

Call:
lm(formula = intensity ~ Condition * Type + Sample_name, data = .)

Residuals:
        1         2         3         4         5         6         7         8         9        10        11 
 0.048650 -0.054058  0.005408 -0.042081  0.023951  0.018130 -0.048650  0.054058 -0.005408  0.042081 -0.023951 
       12 
-0.018130 

Coefficients: (1 not defined because of singularities)
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)           6.560891   0.051612 127.119 2.30e-08 ***
ConditionM           -0.783735   0.072991 -10.737 0.000426 ***
TypeTotal            -1.073941   0.051612 -20.808 3.15e-05 ***
Sample_nameG1_2       0.021609   0.063212   0.342 0.749662    
Sample_nameG1_3      -0.093305   0.063212  -1.476 0.213966    
Sample_nameM_1       -0.020733   0.063212  -0.328 0.759371    
Sample_nameM_2       -0.001057   0.063212  -0.017 0.987465    
Sample_nameM_3              NA         NA      NA       NA    
ConditionM:TypeTotal  0.648356   0.072991   8.883 0.000887 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.06321 on 4 degrees of freedom
Multiple R-squared:  0.9939,    Adjusted R-squared:  0.9832 
F-statistic: 93.16 on 7 and 4 DF,  p-value: 0.0002897

We can see that the model fits the data well (“Multiple R-squared: 0.9673, Adjusted R-squared: 0.955”). We can see that the interaction term that we’re interested in (for changes in RNA binding) significantly deviates from zero in both models.

Below, we make a function to run the linear models on a protein, select the best model and then return the required values from the model. When we run on the same protein as above, we can see that the best model is the one including the TMT tag as a co-variate.

testModels <- function(obj, coeff_of_interest="conditionG1:typeOOPS"){
  
  fit1 <- obj %>% lm(formula=intensity ~ condition + type + condition*type + sample)
  
  fit2 <- obj %>% lm(formula=intensity ~ condition + type + condition*type)
  
  if ( AIC(fit1) < AIC(fit2) ) {
    chosen_fit <- fit1
    fit_name <- "With_tag"
  } else {
    chosen_fit <- fit2
    fit_name <- "Without_tag"
  }
  fit_values <- c(round(coef(summary(chosen_fit))[coeff_of_interest,], 4),
                  round(summary(chosen_fit)$adj.r.squared, 4))
  
  names(fit_values) <- c("lm_fold_change", "lm_std_error", "lm_t_value", "lm_p_value", "lm_adj_R_squared")
  fit_values <- as.data.frame(t(fit_values), stringsAsFactors=FALSE)
  fit_values$fit <- fit_name
  
  return(fit_values)
}
combined_exprs %>% filter(protein == 'A0AVT1') %>% testModels()

Below, we make a function to run the testModels() function on all proteins in turn using dplyr.

runLM <- function(obj, coeff_of_interest="conditionG1:typeOOPS"){
  results <- obj %>%
    group_by(protein) %>% # group the data frame by the unique protein values
    do(testModels(., coeff_of_interest)) # Apply the testModels functions to each group; "." here is the whole data frame 
  results$lm_BH <- p.adjust(results$lm_p_value, method="BH")# FDR 
  
  return(results)
   
}
M_G1 <- combined_exprs %>% runLM()

|=================                                                                            | 19% ~8 s remaining     
|==================                                                                           | 20% ~8 s remaining     
|==================                                                                           | 20% ~9 s remaining     
|==================                                                                           | 20% ~9 s remaining     
|===================                                                                          | 21% ~9 s remaining     
|===================                                                                          | 21% ~9 s remaining     
|===================                                                                          | 21% ~9 s remaining     
|====================                                                                         | 22% ~9 s remaining     
|====================                                                                         | 22% ~9 s remaining     
|=====================                                                                        | 23% ~8 s remaining     
|=====================                                                                        | 23% ~8 s remaining     
|=====================                                                                        | 23% ~8 s remaining     
|======================                                                                       | 24% ~8 s remaining     
|======================                                                                       | 24% ~8 s remaining     
|======================                                                                       | 24% ~8 s remaining     
|=======================                                                                      | 25% ~8 s remaining     
|=======================                                                                      | 25% ~9 s remaining     
|=======================                                                                      | 26% ~9 s remaining     
|========================                                                                     | 26% ~9 s remaining     
|========================                                                                     | 26% ~9 s remaining     
|========================                                                                     | 27% ~9 s remaining     
|=========================                                                                    | 27% ~9 s remaining     
|=========================                                                                    | 27% ~9 s remaining     
|=========================                                                                    | 28% ~9 s remaining     
|==========================                                                                   | 28% ~9 s remaining     
|==========================                                                                   | 28% ~9 s remaining     
|==========================                                                                   | 29% ~9 s remaining     
|===========================                                                                  | 29% ~8 s remaining     
|===========================                                                                  | 29% ~8 s remaining     
|===========================                                                                  | 30% ~8 s remaining     
|============================                                                                 | 30% ~8 s remaining     
|============================                                                                 | 31% ~8 s remaining     
|============================                                                                 | 31% ~8 s remaining     
|=============================                                                                | 31% ~8 s remaining     
|=============================                                                                | 32% ~8 s remaining     
|=============================                                                                | 32% ~8 s remaining     
|==============================                                                               | 32% ~8 s remaining     
|==============================                                                               | 33% ~8 s remaining     
|==============================                                                               | 33% ~8 s remaining     
|==============================                                                               | 33% ~8 s remaining     
|===============================                                                              | 34% ~8 s remaining     
|===============================                                                              | 34% ~8 s remaining     
|===============================                                                              | 34% ~8 s remaining     
|================================                                                             | 35% ~8 s remaining     
|================================                                                             | 35% ~8 s remaining     
|================================                                                             | 35% ~8 s remaining     
|=================================                                                            | 36% ~8 s remaining     
|=================================                                                            | 36% ~8 s remaining     
|=================================                                                            | 36% ~8 s remaining     
|==================================                                                           | 37% ~8 s remaining     
|==================================                                                           | 37% ~8 s remaining     
|==================================                                                           | 38% ~8 s remaining     
|===================================                                                          | 38% ~8 s remaining     
|===================================                                                          | 38% ~8 s remaining     
|===================================                                                          | 39% ~8 s remaining     
|====================================                                                         | 39% ~8 s remaining     
|====================================                                                         | 39% ~8 s remaining     
|====================================                                                         | 40% ~8 s remaining     
|=====================================                                                        | 40% ~8 s remaining     
|=====================================                                                        | 40% ~8 s remaining     
|=====================================                                                        | 41% ~8 s remaining     
|======================================                                                       | 41% ~8 s remaining     
|======================================                                                       | 41% ~8 s remaining     
|======================================                                                       | 41% ~8 s remaining     
|======================================                                                       | 42% ~8 s remaining     
|=======================================                                                      | 42% ~8 s remaining     
|=======================================                                                      | 43% ~8 s remaining     
|=======================================                                                      | 43% ~8 s remaining     
|========================================                                                     | 43% ~7 s remaining     
|========================================                                                     | 44% ~7 s remaining     
|=========================================                                                    | 44% ~7 s remaining     
|=========================================                                                    | 44% ~7 s remaining     
|=========================================                                                    | 45% ~7 s remaining     
|==========================================                                                   | 45% ~7 s remaining     
|==========================================                                                   | 46% ~7 s remaining     
|==========================================                                                   | 46% ~7 s remaining     
|===========================================                                                  | 47% ~7 s remaining     
|===========================================                                                  | 47% ~7 s remaining     
|===========================================                                                  | 47% ~7 s remaining     
|============================================                                                 | 48% ~7 s remaining     
|============================================                                                 | 48% ~7 s remaining     
|=============================================                                                | 49% ~7 s remaining     
|=============================================                                                | 49% ~7 s remaining     
|=============================================                                                | 49% ~7 s remaining     
|==============================================                                               | 50% ~7 s remaining     
|==============================================                                               | 50% ~7 s remaining     
|===============================================                                              | 51% ~7 s remaining     
|===============================================                                              | 51% ~7 s remaining     
|===============================================                                              | 51% ~7 s remaining     
|================================================                                             | 52% ~7 s remaining     
|================================================                                             | 52% ~7 s remaining     
|================================================                                             | 52% ~7 s remaining     
|=================================================                                            | 53% ~7 s remaining     
|=================================================                                            | 53% ~7 s remaining     
|=================================================                                            | 53% ~7 s remaining     
|==================================================                                           | 54% ~6 s remaining     
|==================================================                                           | 54% ~6 s remaining     
|==================================================                                           | 54% ~6 s remaining     
|==================================================                                           | 54% ~6 s remaining     
|==================================================                                           | 55% ~6 s remaining     
|===================================================                                          | 55% ~6 s remaining     
|===================================================                                          | 55% ~6 s remaining     
|===================================================                                          | 56% ~6 s remaining     
|====================================================                                         | 56% ~6 s remaining     
|====================================================                                         | 56% ~6 s remaining     
|====================================================                                         | 57% ~6 s remaining     
|=====================================================                                        | 57% ~6 s remaining     
|=====================================================                                        | 58% ~6 s remaining     
|=====================================================                                        | 58% ~6 s remaining     
|======================================================                                       | 58% ~6 s remaining     
|======================================================                                       | 59% ~6 s remaining     
|=======================================================                                      | 59% ~6 s remaining     
|=======================================================                                      | 60% ~6 s remaining     
|=======================================================                                      | 60% ~6 s remaining     
|========================================================                                     | 61% ~6 s remaining     
|========================================================                                     | 61% ~6 s remaining     
|=========================================================                                    | 61% ~5 s remaining     
|=========================================================                                    | 62% ~5 s remaining     
|=========================================================                                    | 62% ~5 s remaining     
|==========================================================                                   | 63% ~5 s remaining     
|==========================================================                                   | 63% ~5 s remaining     
|==========================================================                                   | 63% ~5 s remaining     
|===========================================================                                  | 64% ~5 s remaining     
|===========================================================                                  | 64% ~5 s remaining     
|===========================================================                                  | 64% ~5 s remaining     
|===========================================================                                  | 64% ~5 s remaining     
|============================================================                                 | 65% ~5 s remaining     
|============================================================                                 | 65% ~5 s remaining     
|=============================================================                                | 66% ~5 s remaining     
|=============================================================                                | 66% ~5 s remaining     
|==============================================================                               | 67% ~5 s remaining     
|==============================================================                               | 67% ~5 s remaining     
|==============================================================                               | 68% ~5 s remaining     
|===============================================================                              | 68% ~5 s remaining     
|===============================================================                              | 68% ~4 s remaining     
|================================================================                             | 69% ~4 s remaining     
|================================================================                             | 69% ~4 s remaining     
|================================================================                             | 70% ~4 s remaining     
|=================================================================                            | 70% ~4 s remaining     
|=================================================================                            | 70% ~4 s remaining     
|=================================================================                            | 71% ~4 s remaining     
|==================================================================                           | 71% ~4 s remaining     
|==================================================================                           | 72% ~4 s remaining     
|===================================================================                          | 72% ~4 s remaining     
|===================================================================                          | 73% ~4 s remaining     
|====================================================================                         | 73% ~4 s remaining     
|====================================================================                         | 74% ~4 s remaining     
|====================================================================                         | 74% ~4 s remaining     
|=====================================================================                        | 74% ~4 s remaining     
|=====================================================================                        | 75% ~4 s remaining     
|=====================================================================                        | 75% ~3 s remaining     
|======================================================================                       | 76% ~3 s remaining     
|======================================================================                       | 76% ~3 s remaining     
|=======================================================================                      | 76% ~3 s remaining     
|=======================================================================                      | 77% ~3 s remaining     
|=======================================================================                      | 77% ~3 s remaining     
|========================================================================                     | 78% ~3 s remaining     
|========================================================================                     | 78% ~3 s remaining     
|========================================================================                     | 78% ~3 s remaining     
|=========================================================================                    | 79% ~3 s remaining     
|=========================================================================                    | 79% ~3 s remaining     
|==========================================================================                   | 80% ~3 s remaining     
|==========================================================================                   | 80% ~3 s remaining     
|==========================================================================                   | 80% ~3 s remaining     
|===========================================================================                  | 81% ~3 s remaining     
|===========================================================================                  | 81% ~3 s remaining     
|===========================================================================                  | 82% ~3 s remaining     
|============================================================================                 | 82% ~3 s remaining     
|============================================================================                 | 82% ~2 s remaining     
|=============================================================================                | 83% ~2 s remaining     
|=============================================================================                | 83% ~2 s remaining     
|=============================================================================                | 84% ~2 s remaining     
|==============================================================================               | 84% ~2 s remaining     
|==============================================================================               | 85% ~2 s remaining     
|===============================================================================              | 85% ~2 s remaining     
|===============================================================================              | 85% ~2 s remaining     
|===============================================================================              | 86% ~2 s remaining     
|================================================================================             | 86% ~2 s remaining     
|================================================================================             | 87% ~2 s remaining     
|=================================================================================            | 87% ~2 s remaining     
|=================================================================================            | 88% ~2 s remaining     
|=================================================================================            | 88% ~2 s remaining     
|==================================================================================           | 88% ~2 s remaining     
|==================================================================================           | 89% ~2 s remaining     
|===================================================================================          | 89% ~1 s remaining     
|===================================================================================          | 90% ~1 s remaining     
|===================================================================================          | 90% ~1 s remaining     
|====================================================================================         | 90% ~1 s remaining     
|====================================================================================         | 91% ~1 s remaining     
|====================================================================================         | 91% ~1 s remaining     
|=====================================================================================        | 91% ~1 s remaining     
|=====================================================================================        | 92% ~1 s remaining     
|=====================================================================================        | 92% ~1 s remaining     
|======================================================================================       | 93% ~1 s remaining     
|======================================================================================       | 93% ~1 s remaining     
|=======================================================================================      | 94% ~1 s remaining     
|=======================================================================================      | 94% ~1 s remaining     
|========================================================================================     | 95% ~1 s remaining     
|========================================================================================     | 95% ~1 s remaining     
|========================================================================================     | 96% ~1 s remaining     
|=========================================================================================    | 96% ~1 s remaining     
|=========================================================================================    | 96% ~0 s remaining     
|==========================================================================================   | 97% ~0 s remaining     
|==========================================================================================   | 97% ~0 s remaining     
|==========================================================================================   | 97% ~0 s remaining     
|==========================================================================================   | 98% ~0 s remaining     
|===========================================================================================  | 98% ~0 s remaining     
|===========================================================================================  | 99% ~0 s remaining     
|============================================================================================ | 99% ~0 s remaining     
|============================================================================================ |100% ~0 s remaining     
|=============================================================================================|100% ~0 s remaining

Below, we plot the p-values. Under the null hypothesis they should show an approximately uniform distribution. If there were a large number of proteins with a significant change in RNA binding, we would expect an additional “spike” with low p-values (<0.05). We see an approximately uniform distribution but with a slight skew towards low p-value. This may indicate the presence of changes in RNA binding but which we are insufficiently powered to detect, e.g low p-value but not significant low p-value.

plotP <- function(obj){
  p <- ggplot(obj, aes(lm_p_value)) + geom_histogram(bins=20)
  print(p)
}
plotP(M_G1)

Below, we tabulate the results. We will use the standard Benjamini-Hochberg method to adjust p-values for the multiple tests we have conducted here (1916 proteins).

summariseSignificantChanges <- function(obj){
  cat("Which fit was best?")
  print(table(obj$fit))
  
  cat("\nHow many p-values < 0.01 per fit 'type'?")
  print(table(obj$fit, obj$lm_p_value<0.01))
  
  cat("\nHow many significant changes in RNA binding (1% FDR)?")
  print(table(ifelse(obj$lm_BH<0.01, "Sig.", "Not Sig."), ifelse(obj$lm_fold_change>0, "Up", "Down")))
  
  cat("\nHow many significant changes in RNA binding (10% FDR)?")
  print(table(ifelse(obj$lm_BH<0.1, "Sig.", "Not Sig."), ifelse(obj$lm_fold_change>0, "Up", "Down")))
}
summariseSignificantChanges(M_G1)
Which fit was best?
   With_tag Without_tag 
       1047         869 

How many p-values < 0.01 per fit 'type'?             
              FALSE TRUE
  With_tag      560  487
  Without_tag   480  389

How many significant changes in RNA binding (1% FDR)?          
           Down  Up
  Not Sig.  661 562
  Sig.      344 349

How many significant changes in RNA binding (10% FDR)?          
           Down  Up
  Not Sig.  376 307
  Sig.      629 604

So, we have detected a lot of proteins with a signficant change in RNA binding!!

We can use volcano plots to take a look at the estimated fold changes and associated p-values

M_G1 %>%
  mutate(sig=ifelse(lm_BH<0.01, "Sig.", "Not sig.")) %>% # add "sig" column
  ggplot(aes(x=lm_fold_change, y=-log10(lm_p_value), colour=sig)) +
  geom_point(size=0.25) + 
  scale_colour_manual(values=c("black", cbPalette[6]), name="") + # manually adjust colours
  facet_wrap(~fit) + # separate plots depending on model used
  xlab("Fold change (log2-space)") +
  ylab("-log10(p-value)") +
  guides(colour = guide_legend(override.aes = list(size = 3))) # manually set aesthetics for legend to make points larger

Finally, we save out the results for use in later notebooks

saveRDS(M_G1, "../results/M_G1_changes_in_RNA_binding_linear_model.rds")
LS0tCnRpdGxlOiAiSWRlbnRpZnlpbmcgY2hhbmdlcyBpbiBSTkEgYmluZGluZyB1c2luZyBhIGxpbmVhciBtb2RlbCIKb3V0cHV0OgogIHBkZl9kb2N1bWVudDogZGVmYXVsdAogIGh0bWxfbm90ZWJvb2s6IGRlZmF1bHQKLS0tCkluIHRoaXMgbm90ZWJvb2ssIHdlIHdpbGwgYXBwbHkgdGhlIGxpbmVhciBtb2RlbCBhcHByb2FjaCBwcmVzZW50ZWQgaW4gKGAxX3NpbXBsZV9leGFtcGxlYCkgdG8gYSByZWFsLWxpZmUgZGF0YSBzZXQuCgpgYGB7ciwgbWVzc2FnZT1GQUxTRSwgd2FybmluZz1GQUxTRX0KIyBsb2FkIHBhY2thZ2VzCmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KGJpb2Jyb29tKQpsaWJyYXJ5KE1TbmJhc2UpCgojIHNldCB1cCBzdGFuZGFyZGlzZWQgcGxvdHRpbmcgc2NoZW1lCnRoZW1lX3NldCh0aGVtZV9idyhiYXNlX3NpemUgPSAyMCkgKwogICAgICAgICAgICB0aGVtZShwYW5lbC5ncmlkLm1ham9yPWVsZW1lbnRfYmxhbmsoKSwKICAgICAgICAgICAgICAgICAgcGFuZWwuZ3JpZC5taW5vcj1lbGVtZW50X2JsYW5rKCksCiAgICAgICAgICAgICAgICAgIGFzcGVjdC5yYXRpbz0xKSkKCmNiUGFsZXR0ZSA8LSBjKCIjRTY5RjAwIiwgIiM1NkI0RTkiLCAiIzAwOUU3MyIsICIjRjBFNDQyIiwgIiMwMDcyQjIiLCAiI0Q1NUUwMCIsICIjQ0M3OUE3IiwgIiM5OTk5OTkiKQpgYGAKCldlIHN0YXJ0IGJ5IHJlYWRpbmcgaW4gdGhlIGRhdGEuIE91ciBpbnB1dCBoZXJlIGlzIHRoZSBwcm90ZWluLWxldmVsIHF1YW50aWZpY2F0aW9uIGZvciB0aGUgTm9jb2Rhem9sZSBhcnJlc3QvcmVsZWFzZSBleHBlcmltZW50IGNvbmR1Y3RlZCBmb3IgdGhlIE9PUFMgTkJUIHBhcGVyLiBJbiB0aGlzIGV4cGVyaW1lbnQsIHdlIHdhbnRlZCB0byBhc3Nlc3MgY2hhbmdlcyBpbiBSTkEgYmluZGluZyBpbiBhcnJlc3RlZC9yZWxlYXNlZCBjZWxscy4gVG8gZG8gdGhpcywgd2UgcXVhbnRpZmllZCAidG90YWwiIHByb3RlaW4gYWJ1bmRhbmNlIGFuZCBSTkEtYm91bmQgKGV4dHJhY3RlZCBieSBPT1BTKSBwcm90ZWluIGFidW5kYW5jZS4gVGhlIHBlcHRpZGUtbGV2ZWwgYWJ1bmRhbmNlcyBoYXZlIGJlZW4gYWdncmVnYXRlZCB0byBwcm90ZWluIGxldmVsIGFidW5kYW5jZSBhbmQgY2VudGVyLW1lZGlhbiBub3JtYWxpc2VkLiBQcm90ZWlucyB3aXRoIG1pc3NpbmcgdmFsdWVzIGhhdmUgYmVlbiByZW1vdmVkLgoKYGBge3J9CnRvdGFsX3Byb3RlaW5fcXVhbnQgPC0gcmVhZFJEUygiLi4vcmF3L3RvdGFsX3Jlc19wcm9fYWdnX25vcm0ucmRzIikKcmJwX3Byb3RlaW5fcXVhbnQgPC0gcmVhZFJEUygiLi4vcmF3L3JicF9yZXNfcHJvX2FnZ19ub3JtLnJkcyIpCmBgYAoKVGhlIGlucHV0IGRhdGEgYXJlIGluIE1TblNldHMuIEFzIGEgcmVtaW5kZXIsIHRoZSBgTVNuU2V0YCBjbGFzcyBtaW1pY3MgdGhlIGBFeHByZXNzaW9uU2V0YCBjbGFzcyBhbmQgY29udGFpbnMgMyBtYXRyaWNlczoKMS4gYXNzYXkgZGF0YSAob2J0YWluZWQgdmlhOiBgZXhwcnNgKQoyLiBmZWF0dXJlIGRhdGEgKGBmRGF0YWApCjMuIHBoZW5vdHlwZSBkYXRhIChgcERhdGFgKQoKVGhlIGFzc2F5IGRhdGEgaXMgdGhlIHF1YW50aWZpY2F0aW9uIG9mIHRoZSBmZWF0dXJlcyAoUFNNcy9wZXB0aWRlcy9wcm90ZWlucykgYW5kIGNvbnRhaW5zIG9uZSBjb2x1bW4gcGVyIHNhbXBsZQoKVGhlIGZlYXR1cmUgZGF0YSBkZXNjcmliZXMgZWFjaCBmZWF0dXJlLCBlLmcgcGVwdGlkZSBzZXF1ZW5jZSwgbWFzdGVyIHByb3RlaW4gYWNjZXNzaW9uLCByZXRlbnRpb24gdGltZSBldGMgZXRjCgpUaGUgcGhlbm90eXBlIGRhdGEgZGVzY3JpYmVzIHRoZSBzYW1wbGVzCgpMZXQncyB0YWtlIGEgbG9vayBhdCBvdXIgdG90YWwgcHJvdGVpbiBxdWFudGlmaWNhdGlvbiBkYXRhLiBJZiB3ZSAicHJpbnQiIHRoZSBvYmplY3QsIHdlIGdldCBhIHN1bW1hcnkgaW5jbHVkaW5nIHRoZSBwcm9jZXNzaW5nIHN0ZXBzIHBlcmZvcm1lZC4gU28sIGhlcmUgd2UgaGF2ZSAyNzYxIGZlYXR1cmVzIChwcm90ZWlucykgcXVhbnRpZmllZCBhY3Jvc3MgMTAgc2FtcGxlcy4gVGhlIGB2YXJMYWJlbHNgIGRlc2NyaWJlIHRoZSAiQ29uZGl0aW9uIiwgIlJlcGxpY2F0ZSIgYW5kICJUeXBlIiBmb3IgdGhlc2Ugc2FtcGxlcy4gV2UnbGwgdGFrZSBhIGxvb2sgYXQgdGhlc2UgaW4gbW9yZSBkZXRhaWwgc2hvcnRseS4gV2UgY2FuIHNlZSB0aGF0IHRoZXJlIHdlcmUgb3JpZ2luYWxsIDIwMTcxIGZlYXR1cmVzIHdoaWNoIHdlcmUgY29tYmluZWQgaW50byAxODExMSBmZWF0dXJlcyB1c2luZyBhIHVzZXItZGVmaW5lZCBmdW5jdGlvbi4gVGhpcyB3YXMgdGhlIHN0ZXAgYXQgd2hpY2ggcGVwdGlkZXMgd2l0aCB0aGUgc2FtZSBzZXF1ZW5jZSBidXQgZGlmZmVyZW50IHZhcmlhYmxlIG1vZGlmaWNhdGlvbnMgd2VyZSBhZ2dyZWdhdGVkLiBUaGVuLCB0aGVzZSAxODExMSBmZWF0dXJlcyB3ZXJlIGNvbWJpbmVkIGludG8gdGhlIDI3NjEgZmVhdHVyZXMgKHBlcHRpZGUtPnByb3RlaW4gYWdncmVnYXRpb24pLiBGaW5hbGx5LCB0aGUgZGF0YSB3YXMgY2VudGVyLW1lZGlhbiBub3JtYWxpc2VkIGFuZCBtaXNzaW5nIHZhbHVlcyB3aXRoIGltcHV0ZWQgd2l0aCBrbm4uIApgYGB7cn0KcHJpbnQodG90YWxfcHJvdGVpbl9xdWFudCkKYGBgCgoKCkhlcmUncyB0aGUgdG9wIG9mIHRoZSBhc3NheSBkYXRhCmBgYHtyfQpwcmludChkaW0oZXhwcnModG90YWxfcHJvdGVpbl9xdWFudCkpKQpwcmludChoZWFkKGV4cHJzKHRvdGFsX3Byb3RlaW5fcXVhbnQpLCAyKSkKYGBgCgouLi4gYW5kIHRoZSBhc3NvY2lhdGVkIGZlYXR1cmUgZGF0YS4gTm90aWNlIHRoYXQgdGhlcmUgYXJlIG1hbnkgY29sdW1ucyBpbiB0aGUgZmVhdHVyZSBkYXRhLiBUaGVzZSBhcmUgYWxsIHRoZSBhZGRpdGlvbmFsIGNvbHVtbnMgb3V0cHV0IGZyb20gUEQgaW4gYWRkaXRpb24gdG8gdGhlIHF1YW50aWZpY2F0aW9uLiBUaGV5IGFyZSBhbGwgc3RvcmVkIGhlcmUgaW4gY2FzZSB0aGV5IGFyZSByZXF1aXJlZC4KYGBge3J9CnByaW50KGhlYWQoZkRhdGEodG90YWxfcHJvdGVpbl9xdWFudCksIDIpKQpgYGAKCi4uLiBhbmQgaGVyZSBpcyB0aGUgcGhlbm90eXBlIGRhdGEuIEFzIHdlIGNhbiBzZWUsIHdlIGhhdmUgMyByZXBsaWNhdGVzIGVhY2ggb2YgIk0iLCAiRzEiIGFuZCAiUyIgcGhhc2UsIHBsdXMgYW4gYWRkaXRpb25hbCBDb250cm9sIHNhbXBsZS4gRm9yIG91ciBwdXJwb3Nlcywgd2UncmUgb25seSBnb2luZyB0byBiZSBpbnRlcmVzdGVkIGluIHRoZSBNIGFuZCBHMSBwaGFzZXMgc28gd2UgY2FuIHJlbW92ZSB0aGUgb3RoZXIgZGF0YS4gQm90aCB0aGUgdG90YWwgYW5kIFJCUCBxdWFudGlmaWNhdGlvbiBvYmplY3RzIGhhdmUgdGhlIGV4YWN0IHNhbWUgb3JkZXIKYGBge3J9CnByaW50KHBEYXRhKHRvdGFsX3Byb3RlaW5fcXVhbnQpKQpgYGAKCkJlbG93LCB3ZSBtYWtlIGEgYm9vbGVhbiBmcm9tIENvbmRpdGlvbnMgdG8gc3Vic2V0IHRoZSBzYW1wbGVzIHRvIHRob3NlIHdoaWNoIGFyZSAiTSIgb3IgIkcxIgpgYGB7cn0KIyBpZGVudGlmeSB0aGUgc2FtcGxlcyB3ZSB3YW50IHRvIGtlZXAKc2FtcGxlc190b19rZWVwIDwtIHBEYXRhKHRvdGFsX3Byb3RlaW5fcXVhbnQpJENvbmRpdGlvbiAlaW4lIGMoIk0iLCAiRzEiKQpwcmludChzYW1wbGVzX3RvX2tlZXApCgp0b3RhbF9wcm90ZWluX3F1YW50IDwtIHRvdGFsX3Byb3RlaW5fcXVhbnRbLCBzYW1wbGVzX3RvX2tlZXBdIApyYnBfcHJvdGVpbl9xdWFudCA8LSByYnBfcHJvdGVpbl9xdWFudFssIHNhbXBsZXNfdG9fa2VlcF0KCnByaW50KHBEYXRhKHRvdGFsX3Byb3RlaW5fcXVhbnQpKQpwcmludChwRGF0YShyYnBfcHJvdGVpbl9xdWFudCkpCgoKYGBgCgpUbyBkZXRlY3QgY2hhbmdlcyBpbiBSTkEgYmluZGluZywgd2UgY2FuIG9ubHkgY29uc2lkZXIgUkJQcyB3aGVyZSB3ZSBoYXZlIGFsc28gcXVhbnRpZmllZCB0aGUgdG90YWwgcHJvdGVpbi4gQmVsb3csIHdlIGlkZW50aWZ5IHRoZXNlIGNhc2VzIGJ5IGludGVyc2VjdGluZyB0aGUgcm93bmFtZXMgb2YgZWFjaCBNU25TZXQgKHRoZSBwcm90ZWluIG5hbWVzKQpgYGB7cn0KaW50ZXJzZWN0aW5nX3Byb3RlaW5zIDwtIGludGVyc2VjdChyb3duYW1lcyh0b3RhbF9wcm90ZWluX3F1YW50KSwgcm93bmFtZXMocmJwX3Byb3RlaW5fcXVhbnQpKQoKY2F0KHNwcmludGYoIk91dCBvZiBhIHRvdGFsIG9mICVzIFJCUHMgcXVhbnRpZmllZCxcbndlIGhhdmUgdG90YWwgcHJvdGVpbiBxdWFudGlmaWNhdGlvbiBmb3IgJXMgcHJvdGVpbnMgPSAlcyAlJSIsCiAgICAgICAgICAgIGxlbmd0aChyb3duYW1lcyhyYnBfcHJvdGVpbl9xdWFudCkpLAogICAgICAgICAgICBsZW5ndGgoaW50ZXJzZWN0aW5nX3Byb3RlaW5zKSwKICAgICAgICAgICAgcm91bmQoMTAwKmxlbmd0aChpbnRlcnNlY3RpbmdfcHJvdGVpbnMpL2xlbmd0aChyb3duYW1lcyhyYnBfcHJvdGVpbl9xdWFudCkpLCAyKSkpCmBgYAoKCkJlbG93LCB3ZSBjb252ZXJ0IHRoZSBNU25TZXQgaW50byBhICJ0aWR5IiBmb3JtYXQgYGRhdGEuZnJhbWVgIHVzaW5nIGB0aWR5YApgYGB7cn0KCnRvdGFsX2V4cHJzIDwtIHRvdGFsX3Byb3RlaW5fcXVhbnRbaW50ZXJzZWN0aW5nX3Byb3RlaW5zLF0gJT4lICMgc3Vic2V0IHRvIGludGVyc2VjdGluZyBwcm90ZWlucwogIHRpZHkoYWRkUGhlbm89VFJVRSkgICU+JSAjICJ0aWR5IiB0aGUgb2JqZWN0LCBlLmcgbWFrZSBpdCBpbnRvIGEgdGlkeSBkYXRhIGZvcm1hdCAtLT4gbG9uZwogIG11dGF0ZShpbnRlbnNpdHk9dmFsdWUpICU+JSBkcGx5cjo6c2VsZWN0KC12YWx1ZSkgIyByZW5hbWUgdGhlICJ2YWx1ZSIgY29sdW1uIC0+ICJpbnRlbnNpdHkiCgpgYGAKClRvcCBvZiB0aGUgdG90YWwgcHJvdGVpbiBleHByZXNzaW9uIGBkYXRhLmZyYW1lYC4gU2VlIGhvdyBlYWNoIGludGVuc2l0eSB2YWx1ZSBub3cgaGFzIGl0J3Mgb3duIHJvdyB3aXRoIHRoZSBvdGhlciBjb2x1bW5zIGRlc2NyaWJpbmcgdGhlIGFzc29jaWF0ZWQgYXNwZWN0cyBvZiB0aGUgaW50ZW5zaXR5IHZhbHVlLCBlLmcgdGhlIHByb3RlaW4gYW5kIGV4cGVyaW1lbnRhbCBjb25kaXRpb24KYGBge3J9CnByaW50KGhlYWQodG90YWxfZXhwcnMpKQpgYGAKTm93IHdlIGRvIHRoZSBzYW1lIGZvciB0aGUgUkJQIHF1YW50aWZpY2F0aW9uIGFuZCB0aGVuIGNvbmNhdGVuYXRlIHRoZSB0d28gZGF0YSBmcmFtZXMgdG9nZXRoZXIuCmBgYHtyfQpvb3BzX2V4cHJzIDwtIHJicF9wcm90ZWluX3F1YW50W2ludGVyc2VjdGluZ19wcm90ZWlucyxdICU+JQogIHRpZHkoYWRkUGhlbm89VFJVRSkgICU+JQogIG11dGF0ZShpbnRlbnNpdHk9dmFsdWUpICU+JSBkcGx5cjo6c2VsZWN0KC12YWx1ZSkKCmNvbWJpbmVkX2V4cHJzIDwtIHJiaW5kKHRvdGFsX2V4cHJzLCBvb3BzX2V4cHJzKQpgYGAKCldlIHdhbnQgdG8gdGVsbCBSIHdoaWNoIGlzIHRoZSBvcmRlciBvZiB0aGUgdmFsdWVzIGluIHRoZSBjb25kaXRpb24gYW5kIHR5cGUgY29sdW1ucyBzbyB0aGF0IHRoZSBmb2xkIGNoYW5nZXMgYXJlIGluIHRoZSBleHBlY3RlZCBkaXJlY3Rpb24sIGUuZyBwb3NpdGl2ZSA9IGhpZ2hlciBpbiBHMSB2cyBNLgpgYGB7cn0KY29tYmluZWRfZXhwcnMkY29uZGl0aW9uIDwtIGZhY3Rvcihjb21iaW5lZF9leHBycyRDb25kaXRpb24sIGxldmVscz1jKCJNIiwgIkcxIikpCmNvbWJpbmVkX2V4cHJzJHR5cGUgPC0gZmFjdG9yKGNvbWJpbmVkX2V4cHJzJFR5cGUsIGxldmVscz1jKCJUb3RhbCIsICJPT1BTIikpCgpgYGAKCgoKTm93IHdlIG1vZGVsIHRoZSBwcm90ZWluIGludGVuc2l0eSBhY2NvcmRpbmcgdG8gdGhlIG1vZGVscyBkZXNjcmliZWQgaW4gYDFfc2ltcGxlX2V4YW1wbGVfdmQuUm1kYC4gQXMgYW4gZXhhbXBsZSwgbGV0J3Mgc2VlIHRoZSByZXN1bHRzIGZyb20ganVzdCBhcHBseWluZyB0aGUgbW9kZWxzIHRvIGEgc2luZ2xlIFVuaXByb3RJRC4KYGBge3J9CmNvbWJpbmVkX2V4cHJzICU+JSBmaWx0ZXIocHJvdGVpbiA9PSAnQTBBVlQxJykgJT4lCiAgZ2dwbG90KGFlcyhDb25kaXRpb24sIGludGVuc2l0eSwgZ3JvdXA9MSwgY29sb3VyPWZhY3RvcihTYW1wbGVfbmFtZSkpKSArCiAgZ2VvbV9wb2ludChzaXplPTIpICsKICBzdGF0X3N1bW1hcnkoZ2VvbT0ibGluZSIpICsKICBzY2FsZV9jb2xvdXJfZGlzY3JldGUobmFtZT0iVGFnIikgKwogIHhsYWIoIiIpICsKICBmYWNldF93cmFwKH50eXBlKQoKZml0IDwtIGNvbWJpbmVkX2V4cHJzICU+JSBmaWx0ZXIocHJvdGVpbiA9PSAnQTBBVlQxJykgJT4lCiAgbG0oZm9ybXVsYT1pbnRlbnNpdHl+Q29uZGl0aW9uKlR5cGUpCgpwcmludChzdW1tYXJ5KGZpdCkpCgoKZml0IDwtIGNvbWJpbmVkX2V4cHJzICU+JSBmaWx0ZXIocHJvdGVpbiA9PSAnQTBBVlQxJykgJT4lCiAgbG0oZm9ybXVsYT1pbnRlbnNpdHl+Q29uZGl0aW9uKlR5cGUrU2FtcGxlX25hbWUpCgpwcmludChzdW1tYXJ5KGZpdCkpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IHRoZSBtb2RlbCBmaXRzIHRoZSBkYXRhIHdlbGwgKCJNdWx0aXBsZSBSLXNxdWFyZWQ6ICAwLjk2NzMsCUFkanVzdGVkIFItc3F1YXJlZDogIDAuOTU1ICIpLiBXZSBjYW4gc2VlIHRoYXQgdGhlIGludGVyYWN0aW9uIHRlcm0gdGhhdCB3ZSdyZSBpbnRlcmVzdGVkIGluIChmb3IgY2hhbmdlcyBpbiBSTkEgYmluZGluZykgc2lnbmlmaWNhbnRseSBkZXZpYXRlcyBmcm9tIHplcm8gaW4gYm90aCBtb2RlbHMuCgpCZWxvdywgd2UgbWFrZSBhIGZ1bmN0aW9uIHRvIHJ1biB0aGUgbGluZWFyIG1vZGVscyBvbiBhIHByb3RlaW4sIHNlbGVjdCB0aGUgYmVzdCBtb2RlbCBhbmQgdGhlbiByZXR1cm4gdGhlIHJlcXVpcmVkIHZhbHVlcyBmcm9tIHRoZSBtb2RlbC4gV2hlbiB3ZSBydW4gb24gdGhlIHNhbWUgcHJvdGVpbiBhcyBhYm92ZSwgd2UgY2FuIHNlZSB0aGF0IHRoZSBiZXN0IG1vZGVsIGlzIHRoZSBvbmUgaW5jbHVkaW5nIHRoZSBUTVQgdGFnIGFzIGEgY28tdmFyaWF0ZS4KYGBge3J9CnRlc3RNb2RlbHMgPC0gZnVuY3Rpb24ob2JqLCBjb2VmZl9vZl9pbnRlcmVzdD0iY29uZGl0aW9uRzE6dHlwZU9PUFMiKXsKICAKICBmaXQxIDwtIG9iaiAlPiUgbG0oZm9ybXVsYT1pbnRlbnNpdHkgfiBjb25kaXRpb24gKyB0eXBlICsgY29uZGl0aW9uKnR5cGUgKyBzYW1wbGUpCiAgCiAgZml0MiA8LSBvYmogJT4lIGxtKGZvcm11bGE9aW50ZW5zaXR5IH4gY29uZGl0aW9uICsgdHlwZSArIGNvbmRpdGlvbip0eXBlKQogIAogIGlmICggQUlDKGZpdDEpIDwgQUlDKGZpdDIpICkgewogICAgY2hvc2VuX2ZpdCA8LSBmaXQxCiAgICBmaXRfbmFtZSA8LSAiV2l0aF90YWciCgogIH0gZWxzZSB7CiAgICBjaG9zZW5fZml0IDwtIGZpdDIKICAgIGZpdF9uYW1lIDwtICJXaXRob3V0X3RhZyIKICB9CgogIGZpdF92YWx1ZXMgPC0gYyhyb3VuZChjb2VmKHN1bW1hcnkoY2hvc2VuX2ZpdCkpW2NvZWZmX29mX2ludGVyZXN0LF0sIDQpLAogICAgICAgICAgICAgICAgICByb3VuZChzdW1tYXJ5KGNob3Nlbl9maXQpJGFkai5yLnNxdWFyZWQsIDQpKQogIAogIG5hbWVzKGZpdF92YWx1ZXMpIDwtIGMoImxtX2ZvbGRfY2hhbmdlIiwgImxtX3N0ZF9lcnJvciIsICJsbV90X3ZhbHVlIiwgImxtX3BfdmFsdWUiLCAibG1fYWRqX1Jfc3F1YXJlZCIpCiAgZml0X3ZhbHVlcyA8LSBhcy5kYXRhLmZyYW1lKHQoZml0X3ZhbHVlcyksIHN0cmluZ3NBc0ZhY3RvcnM9RkFMU0UpCiAgZml0X3ZhbHVlcyRmaXQgPC0gZml0X25hbWUKICAKICByZXR1cm4oZml0X3ZhbHVlcykKfQoKYGBgCgpgYGB7cn0KY29tYmluZWRfZXhwcnMgJT4lIGZpbHRlcihwcm90ZWluID09ICdBMEFWVDEnKSAlPiUgdGVzdE1vZGVscygpCmBgYAoKQmVsb3csIHdlIG1ha2UgYSBmdW5jdGlvbiB0byBydW4gdGhlIGB0ZXN0TW9kZWxzKClgIGZ1bmN0aW9uIG9uIGFsbCBwcm90ZWlucyBpbiB0dXJuIHVzaW5nIGRwbHlyLgpgYGB7cn0KcnVuTE0gPC0gZnVuY3Rpb24ob2JqLCBjb2VmZl9vZl9pbnRlcmVzdD0iY29uZGl0aW9uRzE6dHlwZU9PUFMiKXsKICByZXN1bHRzIDwtIG9iaiAlPiUKICAgIGdyb3VwX2J5KHByb3RlaW4pICU+JSAjIGdyb3VwIHRoZSBkYXRhIGZyYW1lIGJ5IHRoZSB1bmlxdWUgcHJvdGVpbiB2YWx1ZXMKICAgIGRvKHRlc3RNb2RlbHMoLiwgY29lZmZfb2ZfaW50ZXJlc3QpKSAjIEFwcGx5IHRoZSB0ZXN0TW9kZWxzIGZ1bmN0aW9ucyB0byBlYWNoIGdyb3VwOyAiLiIgaGVyZSBpcyB0aGUgd2hvbGUgZGF0YSBmcmFtZSAKCiAgcmVzdWx0cyRsbV9CSCA8LSBwLmFkanVzdChyZXN1bHRzJGxtX3BfdmFsdWUsIG1ldGhvZD0iQkgiKSMgRkRSIAogIAogIHJldHVybihyZXN1bHRzKQogICAKfQoKTV9HMSA8LSBjb21iaW5lZF9leHBycyAlPiUgcnVuTE0oKQoKYGBgCgpCZWxvdywgd2UgcGxvdCB0aGUgcC12YWx1ZXMuIFVuZGVyIHRoZSBudWxsIGh5cG90aGVzaXMgdGhleSBzaG91bGQgc2hvdyBhbiBhcHByb3hpbWF0ZWx5IHVuaWZvcm0gZGlzdHJpYnV0aW9uLiBJZiB0aGVyZSB3ZXJlIGEgbGFyZ2UgbnVtYmVyIG9mIHByb3RlaW5zIHdpdGggYSBzaWduaWZpY2FudCBjaGFuZ2UgaW4gUk5BIGJpbmRpbmcsIHdlIHdvdWxkIGV4cGVjdCBhbiBhZGRpdGlvbmFsICJzcGlrZSIgd2l0aCBsb3cgcC12YWx1ZXMgKDwwLjA1KS4gV2Ugc2VlIGFuIGFwcHJveGltYXRlbHkgdW5pZm9ybSBkaXN0cmlidXRpb24gYnV0IHdpdGggYSBzbGlnaHQgc2tldyB0b3dhcmRzIGxvdyBwLXZhbHVlLiBUaGlzIG1heSBpbmRpY2F0ZSB0aGUgcHJlc2VuY2Ugb2YgY2hhbmdlcyBpbiBSTkEgYmluZGluZyBidXQgd2hpY2ggd2UgYXJlIGluc3VmZmljaWVudGx5IHBvd2VyZWQgdG8gZGV0ZWN0LCBlLmcgbG93IHAtdmFsdWUgYnV0IG5vdCBzaWduaWZpY2FudCBsb3cgcC12YWx1ZS4KYGBge3J9CnBsb3RQIDwtIGZ1bmN0aW9uKG9iail7CiAgcCA8LSBnZ3Bsb3Qob2JqLCBhZXMobG1fcF92YWx1ZSkpICsgZ2VvbV9oaXN0b2dyYW0oYmlucz0yMCkKICBwcmludChwKQp9CgpwbG90UChNX0cxKQpgYGAgCgoKQmVsb3csIHdlIHRhYnVsYXRlIHRoZSByZXN1bHRzLiBXZSB3aWxsIHVzZSB0aGUgc3RhbmRhcmQgQmVuamFtaW5pLUhvY2hiZXJnIG1ldGhvZCB0byBhZGp1c3QgcC12YWx1ZXMgZm9yIHRoZSBtdWx0aXBsZSB0ZXN0cyB3ZSBoYXZlIGNvbmR1Y3RlZCBoZXJlICgxOTE2IHByb3RlaW5zKS4gCgpgYGB7cn0Kc3VtbWFyaXNlU2lnbmlmaWNhbnRDaGFuZ2VzIDwtIGZ1bmN0aW9uKG9iail7CiAgY2F0KCJXaGljaCBmaXQgd2FzIGJlc3Q/IikKICBwcmludCh0YWJsZShvYmokZml0KSkKICAKICBjYXQoIlxuSG93IG1hbnkgcC12YWx1ZXMgPCAwLjAxIHBlciBmaXQgJ3R5cGUnPyIpCiAgcHJpbnQodGFibGUob2JqJGZpdCwgb2JqJGxtX3BfdmFsdWU8MC4wMSkpCiAgCiAgY2F0KCJcbkhvdyBtYW55IHNpZ25pZmljYW50IGNoYW5nZXMgaW4gUk5BIGJpbmRpbmcgKDElIEZEUik/IikKICBwcmludCh0YWJsZShpZmVsc2Uob2JqJGxtX0JIPDAuMDEsICJTaWcuIiwgIk5vdCBTaWcuIiksIGlmZWxzZShvYmokbG1fZm9sZF9jaGFuZ2U+MCwgIlVwIiwgIkRvd24iKSkpCiAgCiAgY2F0KCJcbkhvdyBtYW55IHNpZ25pZmljYW50IGNoYW5nZXMgaW4gUk5BIGJpbmRpbmcgKDEwJSBGRFIpPyIpCiAgcHJpbnQodGFibGUoaWZlbHNlKG9iaiRsbV9CSDwwLjEsICJTaWcuIiwgIk5vdCBTaWcuIiksIGlmZWxzZShvYmokbG1fZm9sZF9jaGFuZ2U+MCwgIlVwIiwgIkRvd24iKSkpCgp9CgpzdW1tYXJpc2VTaWduaWZpY2FudENoYW5nZXMoTV9HMSkKYGBgClNvLCB3ZSBoYXZlIGRldGVjdGVkIGEgbG90IG9mIHByb3RlaW5zIHdpdGggYSBzaWduZmljYW50IGNoYW5nZSBpbiBSTkEgYmluZGluZyEhCgpXZSBjYW4gdXNlIHZvbGNhbm8gcGxvdHMgdG8gdGFrZSBhIGxvb2sgYXQgdGhlIGVzdGltYXRlZCBmb2xkIGNoYW5nZXMgYW5kIGFzc29jaWF0ZWQgcC12YWx1ZXMKYGBge3J9Ck1fRzEgJT4lCiAgbXV0YXRlKHNpZz1pZmVsc2UobG1fQkg8MC4wMSwgIlNpZy4iLCAiTm90IHNpZy4iKSkgJT4lICMgYWRkICJzaWciIGNvbHVtbgogIGdncGxvdChhZXMoeD1sbV9mb2xkX2NoYW5nZSwgeT0tbG9nMTAobG1fcF92YWx1ZSksIGNvbG91cj1zaWcpKSArCiAgZ2VvbV9wb2ludChzaXplPTAuMjUpICsgCiAgc2NhbGVfY29sb3VyX21hbnVhbCh2YWx1ZXM9YygiYmxhY2siLCBjYlBhbGV0dGVbNl0pLCBuYW1lPSIiKSArICMgbWFudWFsbHkgYWRqdXN0IGNvbG91cnMKICBmYWNldF93cmFwKH5maXQpICsgIyBzZXBhcmF0ZSBwbG90cyBkZXBlbmRpbmcgb24gbW9kZWwgdXNlZAogIHhsYWIoIkZvbGQgY2hhbmdlIChsb2cyLXNwYWNlKSIpICsKICB5bGFiKCItbG9nMTAocC12YWx1ZSkiKSArCiAgZ3VpZGVzKGNvbG91ciA9IGd1aWRlX2xlZ2VuZChvdmVycmlkZS5hZXMgPSBsaXN0KHNpemUgPSAzKSkpICMgbWFudWFsbHkgc2V0IGFlc3RoZXRpY3MgZm9yIGxlZ2VuZCB0byBtYWtlIHBvaW50cyBsYXJnZXIKYGBgCgoKRmluYWxseSwgd2Ugc2F2ZSBvdXQgdGhlIHJlc3VsdHMgZm9yIHVzZSBpbiBsYXRlciBub3RlYm9va3MKYGBge3J9CnNhdmVSRFMoTV9HMSwgIi4uL3Jlc3VsdHMvTV9HMV9jaGFuZ2VzX2luX1JOQV9iaW5kaW5nX2xpbmVhcl9tb2RlbC5yZHMiKQpgYGAKCg==